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The Cassette Review Program (CRP) , developed by The 
American College of Life Underwriters ^ is organized into 10 sections 
corrGsponding to the 10 courses of the American College C.L*U, 
diplcma program* It includes both audio tapes and notebooks. The 
fo^mative evaluation of the CRP carried out in 1971 resulted .a a 
restructuring of both the tapes and notebooks* as well as the 
clarification of selected content material, A summative evaluation of 
CRP courses 2 and was carried out in 1972* The evaluation was 
conducted using a design called "Separate Sample Pretest Posttest," 
to ascertain the effects of the two courses on student perforinance. 
Students, classified as aelf-studying or classroom, were selected at 
randcm to participate in the study ^ In totals 16 teachers and 368 
students took part in the evaluation, A multiple-choice test was used 
to measure student knowledge of the CRP material^ and two of the four 
groups of students also received a questionnaire to obtain data on 
the amount and type of use of the CRP, as well as on the technical 
aspects of the prbgram. The test results were subjected to an 
analysis of variance with an unweighted-means solution. The results 
indicated that students who utiliied CRP, in a self^studying or 
classroom mode, did significantly better than students who did not 
use CRP* The evaluation points up the significance of the CRP as an 
instructional device with technical excellence* The questionnaire 
replies indicated a favorable attitude among the users of the CRP* It 
is recommended that the tests used in the evaluation be incorporated 
as part of the CRP package, (DB) 
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AN EVALUATION OF THE AUDIO t^O^BOOK SYSTEM^ 
Richard S. Andrulis 
The American Collaga of Life Underwriters 
Bryn Mawr^ Pennsylvania 

Introduction 

The Cassette Review Program (CRP) t developed by The American Collgge of Life 
Underwriters^ has bean in limittd use for the year 1971-1972. The systeni was 
intended to fulfill the primary objective of providing a mechanisin for students of 
the American College to prepare for examinations of the Chartered Life Under- 
writer diploma program. 
Development = FoCTative Evaluation 

The CW , which is organised into ten sections corresponding to the ten courses 
of the American College C.L,U* diploma program, includes both audio tapas and note- 
books. The tapes are played on a Qassette tape recorder by the student* accompanied 
by the associated textual material incorporated within the notebooks. 

The CW utilizes a format of presentation of the objectives in the aodompanylng 
response booklet. After the student has read the objective for each assignment of 
the C*L,U, itudy guide, he is asked to turn on the cassette recorder* The material 
contained on the cassette tapes ej^lains and describes Information concerning each 
of the stated objectives. At the end of the first explanation for objective one, 
a question is eitter listed in the response booklet or on the tape. The student ■ 
is requested to write out his answer and check his response with a key sheet 
provided in the response booklet* or to listen to the tape for the correct 
answer. The reinforcement of this audio-programmed instruction' method provides 

"^Paper Presentid at toe Annual Meeting of Uie Marican Educational Research 
Association, New Orleans, Louisiana, 1973. 



a mechanism whereby information is presented and asked for with a writtGn confirma- 
tion of the student's response. At times a elarif ieatiqn is eriven to the student 
after he has eonfirmed the answer he has written* The intent of tho clarification 
is to provide further insight into the answer. 

Furthermore, each objective is tested by only one quostion in tho rosponso 
booklet. Since the nature of the CRP is for review only^ and not for initial 
learning, it was felt that no need existed for extensive questioning of the student. 
No norms are provided to the student as to the degree of proficiency in answering 
the questions from the tape or response booklet. The rationale for this is the 
summary and review nature of the CRP, rather than that of a total and thorough 
learning system* 

The logic of the evaluation system dicated that any efforts toward' evaluation 
of the CRP should incorporate the two concepts of formative and summative evaluation 
A plan was devised to answer the questions raised in both the formative and summativ 
mode during 1971-1972, 

The formative evaluation was implemented by the CRP developers durino the 
suraner and fall of 1971 (Langdon, 1971), Information accrued from both content 
judges and field participants resulted in a restructuring of both the tapes and 
notebooks as well as the Glarification of selected content material. Further 
results indicated that the quality of voices, clarity of instructions, presentation 
and format of material were relatively frae of major problems* The results 
forthcoming from this initial formative step indicated that the objectives and 
goals of the CM were met. However, what portion of the final C*L,U. eKamination 
soora is due to the effects of the CM remained unanswered* 
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Sumrnat i ve Evaluation 

During the spring of 1972 an evaluation was carried out on the CRP for aoursos 

2 and 4, The Gvaluation was conducted using a desiqn entitled ''Separate Sample 

Pretest Posttest" (Campbell and Stanley^ E?cperiinental and Quasi-EKperimental Design 

for Research^ 1963) , This design was utilized for the purposes of ascertaining the 

effects of the CKP for Course 2 1 Course 4 on C.L,U. student performancQ. This 

design incorporated the use of four conditions, These conditions are as followsi 

Condition h Students - Received a pretest and a Cassette Review Program, 

but no posttest. 

Condition B students = Received no pretest, but received a Cassette Review 

Program and a posttest* 

Condition C Students - Received a pretest only. 

Condition D Students - Received a posttest only* 
However^ realizing there are two methods of preparation for CL.U. courses , the 
design was modified to aceommodate for this key element. The accommodation of the 
design included the replication of the A, B, C and D Jonditions, for both the 
self-studying students and the elassroom students. 

Methodology 

To accommodate both self=studying and classroom students in this study ^ a 
procedure was implemented to identify the studsnts in one of these two categories. 
Utilizing the racerds of the American College, self--studying students and classroom 
students were selected at random* Classroom students were selected at random by 
a process of Identifying all teachers of Course 2 and Course 4 during the spring 
of 1972, From the list of all available teachers for these two courses, 16 teachers 
were idantlfiidi 8 four Course 2 and 8 for CQursa 4. The process of selection 
of these teachers was stratified according to the dimension of nuniser of students 
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in class. Although the range of students in class was Gxtensive, ranginq from a 
low of 10 to a hiqh of 50 students, only those teachers who had 25 to 35 sfeudlnts 
in their class were selected. This provided not only a large sample for the 
ovaluation but also controlled for the variable of size of class* 

Th& selection of self-studying students proceeded with the identification 
nf the 15 largest cities in the continental United States. From these 15 cities^ 
all self ^studying students who resided within a one hour drive of the city were 
identified- From the total list ©f self-^studylng students studying Course 2 or 
Course 4, a list was generated by random selection method. At this time, teachers 
and self-studying students for all four conditions of the design were identified. 

The next phase was the random distribution of the 8 Course 2 and 8 Course 
4 teachers into condition A, B, Q, or D. The result was that 2 teachers for each 
course^ for each condition, were identified, Self=studying students were also 
randomly distributed along the four conditions of the design , for each course. 
Letters were then prepared and sent to the 16 teachers and the self^^tudying 
students • The letter sent to the teachers ana the self-studiers requesting their 
assistance reflected the four Gonditions of the design. The letter to the self-studying 
students also included a postcard for responding to the invitation as well as a 
self-addressed, stamped envelope* For the students who were in the self-studying 
mode who elected to participate in this design, a second follow-up letter was 
sent giving them specific instructions on what they were required to do as 
a participant in this study, ht the same time the second letter was sent, a 
request was placed to send a copy of the CRP to the students in conditions h and B* 

Of the 16 teachers identified for the study, all agreed to participate. 
Of the 651 students identified as self»studiers who were invited to participate, 
223 actually chose to accept thm Invitation according to one of the four 
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conditions given to them. Fifty-five students declined to participate, whilt 
375 studants did not respond, To provide a location for testing the self-studying 
students^ 15 test coordinators were identified, one in each of the 15 major United 
States cities. With some rearrangements, all testing locations were eventually 
secured in time for implementation of the design. In total, 368 classroom or 
self-studying students participated in the Cassette Review Program evaluation. 

The administration of the appropriate test to classroom students was conducted 
by the teachers aGcording to instructions and a time schedule given to them. The 
16 teachers were also given explicit instructions on what to tell their students, 
depending upon their assignment to condition A, B, C, or D. In addition, the 
teachers in conditions A and B were furnished with all CRP material to present 
to their studetits- Self-studying students were tested through 15 regional centers 
estahlished for this purpose. 

The examination used in the COT evaluation as a measure of student knowledge 
of the CRP material was an objective multipla choice test with five ontions. The 
multiple choice tests were adapted and revised from an initial preliminary study 
evaluating the CRP for all ten courses (Langdon, 1971), The Course 2 test 
included 36 items while the Course 4 test included 26 items. 

In addition to the multiple choice tests, an evaluation questionnaire of 
the CRP itself was also preparrd. This questionnaire was included in material 
for all students, in conditions A and B, The purpose of the questionnaire was 
to obtain data on the amount and type of use of the CRP as well as on technical 
aspects of the program. 

Results , 

The results obtained from the eMminationi administerad to both the self- 
studiers and claisroom students were subjected to an Analysis of Variance with an 
unweighted-means solution (Winer, 1962, p. 242), The results indicated that 

O 
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students who utilizad the CBP , whether in a self-studying or classroom mode, did 
significantly better than students who did not use the CRP (F * 14,528, p < *05, 
df 3, 193, Course 2 and F * 5 . 753 , P < . 05 , df 3 , 120 for Course 4), These results 
are not likely to be obtained by chance alone, but rather reflect the important 
educational significance of the CRP. Generally, they indicate that students who 
utilized the CRP perfornied at a signifiGantly higher level on the examinations than 
students who did not use the cm , for both the self-studiers and classroom participants 
The means of the test scores as well as the number of students are presented in 
Table I. 

Insert Table I About Here 

Generally, the results presented in Table I indicate a significant growth 

from the pretest situation to the posttest situation of knowledge as measured by 

the test included in the Cassette Review Program evaluation. 

A second s€« of analyses were then conducted that compared the performance 
of students who utilized the COT and those who did not utilize the CRP but did 
participate in this design^ with the performance of all students who sat for the 
final exam for Courses 2 and 4, administered in June, In Table II, the results 
for the total number of students who sat for C.L.U. examination 2, as well as 
for the self=studiers and classroom students who participated in our evaluation, 
are preaented. 

Insert Table II About Here 

The results obtained from a comparison between the final examination performance 
of students in the four conditions of the evaluation, with the finsfl examination 
.performance of all students, is presented in Table III for Course 2, 



insert Table III About Here 



The results of this table indicate that two of the four groups in condition 
A and B which utilized the CRP did significantly better than the average for all 
students who participated in the final eKamination . Of the four groups in condition 
C and D which did not utilize the CRP, only one group did better than the nornis. 
In total, five of the eight group scores were in the expected diraction. Although 
not as conclusive as one might like, there is certainly an indication that the use 
of the CRP did significantly assist students in obtaining higher scores on the 
final C.L-U. examination. The explanation for why all the comparisons were not 
in the intended direction is as follows. The evaluation included an eKamination 
covering the content of the CRP for both the user and the nonuser of the program. 
The aKajnination is conceivably a learning experience which could have increased 
one's final examination score. 

The results of the comparison between students participating in the evaluation 
with the total group of students taking the Course 4 eKams are presented in 
Table IV and Table V. 



Insert Table IV About Here 



Insert Table V About Here 



The results indicate more positively that students who utilized the CRP in 
conditions A and B did signifiGantly better than all students who participated in 
the exainination of Course 4, for two of four possible cases. On the other hand^ 
students who did not utilize the program in condition C and D did not do significantly 
better than all the students in any of the four cases. These results, in conjunctfon 
with data from Course 2^ indicate and support the hypothesis that the CRP 
did significantly affect scores in a positive direction, for users of the 



'8 

The final results of this analysis were obtained from the CRP Questionnaire, 
These results are presented in Tabla vi for all students who used CRP 2 and 4 
tofothgr. 

insert Table VI About Here 

Digqussibn 

One summary indication forthcoiTiing from the evaluation is not only the significance 
of the CRP as an instructional device, but the technical excellence with which it 
has been developed. The results of the CRP Questionnaire indicate that a favor- 
able attitude exists among the users of the GRP. It is clear that the student will, 
spend both time and money necessary to obtain the CRP as well as a cassette recorder 
to increase their knowledge in the C.L.U, program. The impact of a positive 
attitude on the part of the students can be substantiated with the results of 
this evaluation to indicate the important nature of the GRP. 

One significant recommendation for the Cassette Review Program is the incorporation 
of the tests used in the evaluation as part of the CRP package. The student would 
utilize the instrument after a thorough study of the CRP booklet and tapes* The 
test, along with the rest of the key responses^ could be used as a self ^administered 
diagnostic instrument for the student's own self--evaluation. 
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TABLE I 



Test 
Characteristics 



No. Of Items No. of Students Mean Score 

In Test SS* CR* Total SS CR Total 



COURSE 4 



Group A 

(Pretest + COT) 



26 



29 



21 



50 15.9 15,5 16,3 



Group B 

(CRP + Posttest) 

Group C 
(Pretest) 



26 
26 



21 



11 



15 



29 



36 18.1 18.9 18,5 



40 17.2 13,8 14.5 



Group D 
(Posttest) 



.26 



13 



23 



36 16.5 15.4 15.1 



Course 2 



Group A 

(Pretest + CBP) 



36 



30 



28 



58 23,3 24.6 23,9^ 



Group B 

(CRP + Posttest) 



36 



24 



33 



57 30,7 29.2 30.0 



Group C 
(Pretest) 



36 



39 



45 26.0 24.2 24,4 



Group D 
(Posttest) 



36 



32 



41 24,0 26.5 25.9 



* SS ^ Self-studiers 
CR * Clasiroom 
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TABLE II 
Course 2 



MEAN 



STANDARD 
DEVIATION 



All Students 

Self-Studiers 

Group A 
Group B 
Group C 
Group D 

Classroom 

Group A 
Group B 
Group C 
Group D 



4095 



30 
24 
6 
9 



28 
33 
39 
32 



75.95 



77.5 
82,4 
84.6 
79,7 



82.0 
79.9 
79,5 
78,98 



10,00 



9.59 
7.41 
9.05 
9.94 



10,00 
9,16 
5.48 

11,04 



TABLE III 
COURSE 2 



Self-Studlers Claisroom 



Group A t ^ ,641 P >,05 t ^ 2.92 P <.05 

Groups ' P < ,05 t=1.89 P>,05 



Group C t ^ 1.899 P > ,05 / ' t 3.340 P <.05 

Group D 1,05' P < . 05 t ^1,88 B >.05 
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TABLE IV 
Course 4 

Mean 



Standard 
Devtatlon 



All Students 

Self-Studiers 

Group A 
Group B 
Group C 
Group D 

Classroom 

Group A 
Group B 
Group C 
Group D 



3748 



29 
21 

H 
13 



21 
15 
29 
23 



76.61 



83.6 
78.9 
82. 
78.4 



79.5 
80.6 
79.1 
79.8 



11.31 



47.1 
102.8 

83.5 
115.2 



89,0 
52.9 
96.5 
90.9 
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' TABLE V 
Course 4 

Sslf-Studlire Clasiroom 



Group A t - 5.4 P< .OS t ■ 1.2 P>.05 

Group 3 t - .92 P> .05 t * 2,3 P <.05 



Group C t ■ 1.4 P> ,03 t ■ 1.1 P >,05 

Group D t ■ ,53 .05 t ■ 1.4 P *.0S 
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TABLE VI 



I. Hours per week one used: 
Total hours spint ons 



Cassette h Cassette B Cassatte C 
2.34 2.33 2,32 

4.74 4.56 4.51 



2. Did you tver use thi raviaw 
booklat without the cassette? 



YES 
40% 



If Yes, for which Qassatte? 



33% 



35% 



33% 



3. Whan you ganerally studied, 
did you use the Cassette 
Review Program? 



YES 
41% 



4, How did you use the COT? 

A) Read the material first and then listened" to 

the reeorded aestien 47% 

B) Listened to the racording first and then 

read the material 38% 

C) Read the text and listened without a 

ipioifio order 15% 

Information Transfer 

5, How do you feel this means of information transfer aomparee to studying 
without the use of oassettii? 



Casaettti are 
muoh better 



K ^ aai 



Reading only 
is muoh bitter 



6. Would you oontinue to uii the CKP? 



92% indicated YEi 



7i HiVi you evar uitd a oaiitta profram 
before a§ a means of learning? 

10. Have you evir reoeivid interest 
from friends or assooiates as to 
how they oould uit the COT? 
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301 indicitid YES 



201 indicated VBS 
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TABLE VI - Gontinuid 

chnical Information 

Sound Quality - Reported as Mean Scores for each cassette 



On a scale of 1 (clear) to 5 {fuzzy) 

Caesetta A 1.32 
Cassette B 1.40 
Cassette C 1.41 



Speed of inforaiation - ©n a scale of 1 (too fast) to 5 (too slow) 

Cassette A 2.7S 
Cassette B 2,63 
Cassette C 2.76 



Amount of information - on a soale of 1 (too much) to 5 (too little) 

Cassette A 3.33 
Casiette B 3,20 
Caisetta c 3.25 



Wording of information - on a scale of 1 (clear) to 5 (unclear) 

Caisitte A 1,61 
Cassette B 1,64 
Cassette C 1*80 



Ware you able to obtain a racorder? 1 ^ with difficulty 2 ^ with ease 
The mean is 1,82, 



Did you have to pay for a cassette recorder to use the CRP? 47% indicated Yes 

531 indicated No 
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